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ABSTRACT 
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Abstract 



The use of Bayesian decision theory to solve problems in 
t^st— based decision making is discussed. Four basic decision 
problems are distinguished: selection, mastery, placement and 
classification. For each type of decision, further 
restrictions or generalizations may hold, namely multivariate 
test scores, sequential testing, multiple criteria, multiple 
populations or quota restrictions. In some applications, 
combinations of the basic types of decisions may occur. 
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Applications of Decision Theory to 
Test-based Decision Making 

Historically,, the use of tests has its roots in the necessity 
of selection and placement decisions in education, the army, 
and the public administration. This is excellently 
demonstrated in DuBois's (1970) historiography of such cases 
as Binet's pioneering work on developing a test for the 
assignment of retarded children to special education,, the 
testing of conscripts for placement in the army during World 
War I, and the examination of applicants for the civil 
service in ancient China. It is no coincidence that in each 
of these fields decision making is characterized both by a 
high visibility and a massive number of examinees . In such 
cases it seems logical to use tests to base decisions on. 

Inspired by these early successes, decision makers have 
been using tests ever since. Nowadays, the use of tests has 
pervaded such fields as the admission of students to schools, 
the selection of personnel in public as well as private 
settings, the assignment of clients to therapeutic 
creatments , the choice of careers in vocational guidance 
situations , pass-fail decisions in instructional units , 
certification, personnel review, tracking decisions in 
individualized study systems, and the evaluation of training 
programs. Although their contents and format may vary, it is 
inconceivable that the use of tests will ever leave these 
fields . 

o 
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It is conspicuous that, although the practice of testing 
has its roots lirmly in decision making, test theory has been 
developed mainly as a theory of measurements. This was 
already manifest in Spearman's pioneering work on what is now 
known as classical test theory. In this theory, test scores 
are modeled as a linear combination of a true score and an 
error of measurement and the concern is primarily in 
quantities such as the reliability coefficient, the standard 
error of measurement — as well as in their properties as a 
function of test length., item selection, and the like. Modern 
item response theory shows the same concern with measurement 
(ability estimation), and was not conceived as a theory of 
decision making either. The history of test theory shows a 
few exceptions, though, of which the publication of the 
Taylor-Russell (1939) tables, with their subsequent influence 
on the testing literature, and Cronbach and Gleser's (1965) 
Psychological tests and personnel decisions deserve special 
mention. To date, the latter has been the first and only 
monograph attempting to provide test-based decision making 
with a sound theoretical basis. 

Recently, however, the situation has changed somewhat 
and some test theorists are now seriously involved in 
attempts at modeling and optimizing the use of test scores 
for decision making. A major impetus for this has come from 
the introduction of modern instructional systems as 
individualized instruction learning for mastery , and 
computer-aided instruction. In such systems. testing 
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primarily serves instructional decision making, and an 
important task of their developers is to design optimal 
decision procedures. A seminal paper by Hambleton and Novick 
( 1973 ) was the first to point at the challenge of these 
developments to classical test theory. 

Although test theory has long ignored decision problems, 
at a more abstract level the study of optimal rules for 
decision making has had a long tradition in statistics dating 
back to early publications such as von Neumann and 
Morgenstern (1944) and Wald (1950). More up-to-date 
treatments of statistical decision theory can be found in 
DeGroot (1970) and Ferguson (1967); a short but excellent 
introduction is given in Lindgren (1976, chap. 8). It is the 
primary intention of this paper to demonstrate how the 
various types of decision making problems in testing can be 
solved using the framework of statistical decision theory. In 
particular, in doing so, an (empirical) Bayesian point of 
view will be assumed. In the next sections, first some 
decision-theoretic notions will be introduced. Then, a 
classification of all possible types of test-based decisions 
will be given . Subsequently , the main part of this paper 
showing how problems with respect to the various types of 
decisions can be solved using Bayesian decision theory is 
addressed. In the final section some results will be 
presented on the application of decision theory to the 
problem of simultaneous optimization of combinations of 
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different types of decisions in individualized instruction 
systems. 

Elements of a Statistical Decision Probletr 

Statistical decision problems arise when a decision maker is 
faced with the need to choose a preferred action; the outcome 
of the action depends on the state of nature about which only 
partial information is available, A simple example is the 
vacationer who has to decide whether or not to go for the 
beach, but must rely on a forecast for information about the 
weather;, a more sophisticated one is the researcher who has 
to decide on the basis of sample data which of his/her 
hypotheses holds for a given population. 

The set of all possible states of nature relevant for 
the decision problem is known as the state space in decision 
theory. This set will be denoted by ft whereas a numerical 
parameter 0) will be used to index the 1 individual states 
in O (When SI is discrete, (0 S . s = 1. . ... S. will be used.) 
Let A be the set of all possible actions from which the 
decision maker can choose. Technically, A is known as the 
action space . Individual actions will be denoted by a 

(continuous action space) or a t , t = 1 T, (discrete 

action space). For each action a € A, the decision maker is 
confronted with certain consequences . These consequences 
depend not only on the action chosen but also on the 
(unknown) state 0) e Q nature is in. Some of these 
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consequences may be positive; others negative. It is supposed 
that the decision maker is able to summarize the consequences 
of his/her actions given the true state of nature into an 
evaluation on a numerical scale. As this scale is assumed to 
run from negative to positive evaluations, it is what is 
technically known as a utility scale. So in the following, 
the existence of a utility function u(o),a) on ft x A is 
supposed. For each possible combination of an action and a 
state of nature this function indicates how positively the 
decision maker evaluates the outcomes. When A is discrete, 
the utility function will be notated by u t ((D). 

The decision problem would be easy if the true state of 
nature were known. If nature is in state Oq. the best action 
is the one for which u(o>o.a) is maximal. In most decision 
problems, however, nature does not fully disclose its true 
state; all we have at hand are fallible data — for example, 
information from a sample, or subjective beliefs. A neat way 
of formalizing this is to assume the existence of a random 
variable Z representing the outcome of some experiment or 
measurement conducted to get known the true state of nature, 
and whose distribution depends on co. In the following it is 
assumed that the family of probability distributions of Z 
with distribution functions F(z) s F z (z;C0) is known. 

If the true state of nature is not known with certainty, 
the decisions based on A are likely to be less than optimal. 
A rational approach, then, is to look for a decision rule 
that optimizes the outcomes across repetitions of the same 
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decision problem, but with nature in possibly different 
states. The main purpose of decisior theory is to find such 
rules and to study their properties. Formally. a 
( nonrandomized) decision rule is a prescription specifying 
for each possible value z of Z which action a € A has to be 
taken. Hence, a proper notation is to write the decision rule 
5 as a mapping from the data Z to the action space A: 
A = 5(Z) . Due to the fact that Z is a random variable, using 
8 implies that the actions are also random. At first sight 
this may seem embarrassing. However, there is no way to 
escape the random nature of our data about go. Moreover, the 
decision maker is free to choose whatever rule we would like 
to have from the set of possible mappings from Z to A. which 
gives him the opportunity to select an optimal one. 

It is obvious that our criterion for selecting an 
optimal decision rule should have to do with the utilities of 
the decision outcomes, u(co.a). The function A = 8(Z) implies 
that these utilities must be considered as realizations of a 
random variable U = u(co.8(Z)). In such cases it is natural to 
replace this variable by its expectation. Therefore, we 
define the expected utility as: 



If nature is in state co and decision rule 8 is adopted, then 
(1) shows the utility to be expected. However, the actual 



(1) 



R(co.8) s E[u(co.5(Z))] 
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actions still depend on the values taken by Z, so their 
utility usually will vary about the value of (1). 

It is still not possible to define a criterion for a 
uniformly best decision rule 5 using the expected utility 
function in (1). The reason is its dependency on the unknown 
value (0. One way out of this problem would be to make a 
sensible choice for this value, say i)q. and to select a 8q. 
such that the expected utility R((0q ,8) is maximal . This 
approach is taken, for instance, in minimax theory where (0q 
is selected as a value of CI that represents the least 
favorable state of nature to the decision maker. As a 
consequence, the decision maker is guaranteed that the 
minimum expected utility for 8q is never smaller than 
R(N0' 5 0> . 

Another approach is not to select one special state CI 
of nature, but to assume a distribution function G(co) over 
CI. This may represent the decision maker's subjective 
probabilities of the possible states of nature or its 
empirical distribution. For both interpretations G(co) is 
known as the a priori distribution (or prior) because it 
represents knowledge about (D available before the data Z = z 
are observed. Having G(to). we are now able to define the 
liayes utility of decision rule 8 as 
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(2) 



B(5) a E[R(co,8)] 

= f R(G>.8)dG(o». 



In the literature this quantity is also known as the Bayes 
risk of the decision procedure, although, strictly speaking, 
this name is only proper if a loss function instead of a 
utility function is used. 

The Bayes utility in (2) only depends on the decision 
rule. It now seems obvious to select from the class of 
possible rules the one, say 5*. that maximizes the Bayes 
utility: 

(3) B(8*) = max B(8) . 

8 

Rules satisfying (3) are known as Bayes rules . Throughout 
this paper it will be assumed that the quantities used in the 
aoove definition of a Bayes rule exist for the problem at 
hand (though it will not necessarily be true that the problem 
has a unique Bayes rule) . 

Monotone Bayes rules 

For the actual maximization in (3) it would be helpful if the 
(possibly infinitively) largo set of all possible rules 8 
could be restricted to some subset of a tractable form. 

In addition to this technical consideration, there also 
is a somewhat less lofty reason for which the attention 
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sometimes has to be restricted to a subset of possible rules. 
This has to do with the acceptability of some types of rules 
among those involved in the decision procedure. In education, 
for example , students , teachers and administrators are 
familiar with selection decisions in which the decision rules 
have a monotone form : students are admitted to a program if 
their grade-point average or their test score are above a 
certain cutting point and rejected otherwise. It would mean a 
shock to all parties if some institutions changed the form of 
their selection rule and started, e.g., admitting students 
with low or high scores while rejecting those with 
intermediate ones. However, the restriction to monotone rules 
is only correct if they constitute an essentially complete 
class (Ferguson, 1967, p. 55); otherwise rules with a higher 
expected utility r.re wrongly ignored. 

The conditions under which a class of monotone rules is 
essentially complete are known (Chuan, Chen & Novick, 1981; 
Ferguson, 1967, sect. 6.1). Two conditions have to be met: 
First, the so-called posterior distribution of (0 given Z = z 
should be stochastically increasing, that is, if F(co|z) is 
the distribution function of (0 given Z = z , zi > z2 must 
imply F(co|zi) < F(co|z2) for all o>. Second, there should be 
an ordering of the actions for which the difference between 
the utility functions for adjacent pairs of actions changes 
sign at most once. For the decision problems dealt with in 
this paper,, the conditions will be made more specific below. 
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A Classification of Test-based Decision Making 

The use oi test scores for decision making in education and 
psychology can be classified in a simple way (van der Linden. 
1985a, 1985b). In each of these settings three basic elements 
can be identified and each type of decision making can be 
viewed as a specific configuration of these elements. In 
general , four different types of decision making can be 
distinguished. Further, for each of these types four possible 
restrictions or refinements can apply. This classification of 
test— based decision making will now be elaborated. In the 
next sections it is then shown how Bayesian decision theory 
can be applied to the problems in this classification. 

Basic Elements of Test-based Decisions 

Each type of decision unking can be identified as a specific 
configuration of one or more of the following elements: 

(1) A test that provides the scores on which the decisions 
are based; 

(2) One or more treatments with respect to which the 
decisions are made; 

(3) One or more criteria by which the successes of the 
treatments are measured. 

The term •'test** is used here mainly because of the focus of 
this paper. It could easily be replaced by any other 
measuring instrument or source of data without invalidating 
the content of this paper. Likewise, "treatment" is a generic 
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term here, referring to whatr /er manipulation, experiment, or 
program is used to change the condition of individuals. 
Examples of treatments in educational and psychological 
settings are; instructional programs, applications of AV- 
materials, t psychological therapies. It shoula be noted 
that information about the success of the treatment on the 
criterion may b* provided by any source. In this paper it is 
assumed that the information is quantitative by nature. In 
practice, the criterion is often measured by another test. 

Types of Decisions 

Four basic types of decisions are distinguished. Each type of 
decision can be represented by a unique flowcnart containing 
one or more o - the above elements . 

(1) Se lection decisions . In selection problems the decision 
is the acceptance or rejection of individuals for a 
treatment. A typical feature of the selection decision is 
that the test is administered before the treatment but that 
the criterion is measured afterwards. Well— known examples of 
selection decisions are selection of personnel in industry 
and admission of students to educational programs . Figure 1 
gives a flowchart displaying the structure of the selection 



Insert Figure 1 about here 
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decision. Selection research has had a long tradition in 
educational and psychological testing in which the selection 
decision was viewed as a prediction problem. Until Cronbach 
and Gleser ( 1965) , the usual approach was to establish 
regression lines or expectancy tables to predict criterion 
scores anc to accept individuals with predicted criterion 
scores above a given threshold value . 

(2) Mastery decisions . Unlike selection decisions, mastery 
decisions are made after a treatment. The content of the 
decision is whether or not individuals who followed the 
treatment are successful on the criterion. A further feature 
is that the criterion is internal and not external to the 
test. It is the unreliability of the test as a representation 
of the criterion that creates the mastery decision problem. 
Due to measurement error, the possibility of false-negative 
and false-positive decisions exists, and it is the task of 
the decision maker to rrinimize their consequences. Figure 2 
shows the formal structure of the mastery decision problem. 



Examples of mastery decisions are pass-fail and certification 
decisions in education. 

(3) Placement decisions . In placement problems, several 
alternative treatments are available and it is the decision 



Insert Figure 2 about here 
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maker's task to assign individuals to the most promising 
treatment. The same test is administered to each individual 
and the success of each treatment is measured by the same 
criterion . Placement decisions differ from selection 
decisions by the fact that oiore than one treatment is 
available and that each individual is assigned to a 
treatment. The case of a placement decision with two 
treatments is shown in Figure 3 . Examples of placement 
decisi'us can be found in individualized instruction where 



students are allowed to follow different routes through 
instructional units but , regardless of routes , the same 
criterion is appropriate. Aptitude-treatment interaction 
(ATI) research has given the main impetus to interest in the 
placement problem. In ATI research, the traditional approach 
has been regression analysis with a separate regression line 
for each treatment assigning individuals to the treatment 
with the largest predicted criterion score. 

(4) Classification decision s. As is clear from Figure 4, the 



Insert Figure 3 about here 



Insert Figure 4 about here 
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difference between placement and classification decisions is 
that in the latter , each treatment has its own criterion . 
Further properties of the two types of decisions are equal. 
Examples of classification decisions arise in vocational 
guidance situations where most promising careers or training 
programs must be identified The most popular approach to 
classification decision! has been the use of linear- 
regression techniques. Each criterion is then mapped on a 
common utility scale and the decision rule is to assign 
individuals to the treatment with the largest predicted 
utility. 

Further Restrict ions and Generali zations 

It should be noted that the above types of decisions are not 
always met in their pure forme. These decisions often occur 
in combinations ; also, further restrictions or 
generalizations may apply. An example of a combination of two 
types of decisions arises in a selection problem where the 
criterion is unreliably measured. If success on the criterion 
is defined by a threshold value, then, in fact, after the 
treatment a mastery decision has tobe taken, and the problem 
is a selection-mastery decision problem. A combination of a 
selection and a classification problem is met if more than 
one treatment is available but not all individuals are 
accepted for a treatment. In individualized instruction - for 
instance, as implemented in CAI-systems - decision making can 
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be viewed as guiding students through a network of several of 
the above decisions . The simultaneous optimization of such 
networks will be discussed in the final section of the paper. 

^or each basic type of decision , one or more of the 
fallowing restrictions or generalizations may apply: 

(1) Multivariate test scored. Instead of a single test, a 
battery of tests may be used to base the decision on. The use 
of test batteries has had a long history in the practice of 
personnel selection and vocational guidance. Formally,, the 
use of test batteries implies decision rules defined on a 
vector of test scores instead of a single score. 

(2) Sequential testing . In a sequential testing strategy, 
test items are administered until a decision can be made with 
a desired level of certainty. The recent introduction of the 
computer in educational and psychological testing has 
stimulated the interest in sequential testing strategies for 
decision making. If tests are used in this mode, sequential 
Bayesian procedures can be used (Lindgren, 1976,, sect. 8.5). 

(3) Multiple criteria . In some applications, the success of a 
treatment has to be measured on more than one criterion. Each 
individual criterion is then supposed to reflect a different 
aspect of the treatment. Formally, the presence of multiple 
criteria implies the necessity to define utility functions on 
a vector of criterion scores instead of a single score. 

(4) Multipl e populations . The presence of different popula- 
tions of examinees reacting differently to the test items may 
create the problem of "fair" decision making. In education., 
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the problems of fair selection and mastery decisions have 
been struggled with for a long time, ; in particular for 
populations defined by race or sex. Formally, the presence of 
populations reacting differently to test items implies 
different probability distributions of test and criterion 
scores for each population. In addition, the decision maker 
may have different utilities associated with different 
populations. As a result, a separate decision rule has to be 
established for each population. 

(5) Quota restrictions . So far it has been assumed that the 
number of vacancies in each treatment is free. Due to the 
shortage of resources , however , these numbers may be 
constrained. Consequently, Bayes rules for quota-restricted 
decisions have to be found by methods of constrained 
optimization. 



Conclusion 

The abovo classification of test-based decisions shows four 
basic types of decisions that may occur separately or in 
combination. Further,, for each decision one or more 
refinements or generalizations may hold . In addition,, the 
utility structure and probability distributions may vary from 
problem to problem. However,, the key to finding Bayes rules 
for each possible decision is still the optimization of its 
Bayes utility. This will now be illustrated for a sample of 
decision problems. 
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Selection Decisions with Linear Utility 
. «- 

In selection decisions, test scores are used to decide on the 
acceptance or rejection of individuals for a treatment, with 
success measured on a future criterion. In order to apply the 
framework of Bayesian decision theory, the "unknown state of 
nature*' should now be interpreted as the individual's unknown 
criterion score and the "data" about this state is provided 
by the test score. For a randomly sampled individual, let the 
criterion be a continuous random variable Y, w h as possible 
states, success (Y £ y c ) and failure (Y < y c ). The test score 
is assumed to be a discrete random variable X with possible 

values x = 0 f 1 n (number-right score). The information 

in X on Y is given by a joint probability function k(x,y). 
Since the conditions for a monotone decision are assumed to 
be met, the Bayes rule for the selection problem is a cut— off 
score x c , with an acceptance and a rejection decision for X > 
x c and X < x c , respectively. 

Formally, utility is a function defined on the true 
state of nature with a possibly different form for each 
action. A moment's reflection shows that in the present 
problem utility should be an increasing function of the 
criterion for the acceptance decision, but a decreasing 
function for the rejection decision: The higher the criterion 
score of an accepted individual the higher the utility of 
the decision ; whereas the opposite holds for a rejected 
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individual. A linear utility function that meets this 
property is given in van der Linden and Mellenbergh (1977): 

(bo(yc-y) + ao for x < xc 
bl(y-yc) + a l f° r x > xc . bo, bi > 0 . 

This function, which is shown in Figure 5. consists of two 
additive components: 

(1) bo(y c -y) anc * t>i(y-y c ) represent amounts of utility 
dependent on the difference between the criterion score 
and success threshold y c with constants of 
proportionality bg and b^ ; 

(2) ag and a^ are amounts of utility independent of the 
criterion score but dependent on the decision. They can 
be used, for instance, to allow for treatment costs. 



Insert Figure 5 about here 



When sampling individuals from the population , the Bayes 
utility for decision rule x c is equal to 

x c -l 

(5) R(x c ) = 1 J[b0(yc-y) + ao]k(x,y)dy + 
x=0 
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x=x r 



[bo(y-yc) + al]k(x.y)dy. 



Using k(x.y) = g(y|x)h(x). 



g(y|x)dy = 1 and 



f 



yg(y|x)dy = E(Y|x). it follows 



x c -l 

(6) R(x ) = I {bO(E(Y|x)-y c 3 + ao)h(x) 

c 

x=0 



I {bi(E(Y|x)-y c ] - ao)h(x) 



x=x. 



where E(Y|x) is the regression function of Y on X. 
Completing the first sum 



(7) R(x c ) = Z <bO[E(y|x)-y c ] + aO)h(x) - 

x=0 

n 

X {(bo+bl) [E(Y|x)-y c ] + (ao-ai ) }h(x) . 
x=c 
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Since the first sum is now a constant, bg + bj > 0, h(x) £ 0 
for all x, and the monotonicity conditions guarantee that 
E(Y|x) is increasing in x. (7) is maximal for the value of 
the smallest value of x c for which 

(8) (bQ+bi) [E(Y|x)-y 0 l + (ao+aj) 

is not negative. If the monotonicity conditions are not 
strict or it does not hold that h(x) > 0 in the neighborhood 
of the solution, this value of x may not be unique. 
Throughout this paper it will be assumed that conditions like 
these are fulfilled. 

The regression function ECYjx) can easily be estimated 
by drawing a sample from the population and administering the 
treatment . 

It should be noted that (8) is the difference between 
the two conditional expected utilities given X = x associated 
with the acceptance and rejection decision. This is clear 
from inspection of the bracketed terms in (6) and (7). The 
expectations are known as posterior expected utilities; they 
can be considered the expected utilities after the 
observation X = x has been made. For a monotone decision it 
holds in general that the optimal cut-off score is located at 
the point at which the posterior expected utilities cross 
(e.g.. De Groot. 1970. sect. 8.9). In the following, this 
property will be used without further validation. 
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A closed-form solution exists if the regression function 
is linear, that is, if 

(9) E(Y|x) = Px + a 

Then, it follows from (8) that xc is the smallest value 
larger than 

y c - a ao - ai 

(10) + . 

(3 p(b0+bi) 

An interesting case arises if aQ = a i- Under this condition 
the second term in (10) vanishes and the solution contains 
the regression parameters only. For this and other properties 
of the linear utility function, see van der Linden and 
Mellenbergh (1977). 

Multiple Populations 

As noted earlier, the presence of multiple populations in 
selection decisions creates the problem of fair selection if 
each population reacts differently to the test items. In such 
a case. Zhe test items are often said to be "biased" against 
one *>r more of the populations. 

The only thing needed to deal with multiple populations 
in selection decisions seems to include separate probability 
distributions of test and criterion scores for each 
population in the model to allow for differential item 
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properties. However, the problem of fair selection often 
involves the notion of "disadvantagedness" as well, in 
particular when some of the populations are defined by race 
or sex. As Gross and Su (1975) and Novick and Petersen (1977) 
argue, this aspect of fair selection is only a question of 
utilities. A selection rule is "fair" if those involved in 
the decision process accept the utility structure v erlying 
the decision rule. Hence, in addition to separate p ©ability 
distributions, separate utility functions are needed to allow 
for different utility structures for the populations. 

Now the above selection model can easily be adapted to 
the case of multiple populations (Mellenbergh & van der 

Linden, 1981). Let i = 1 p demote the populations in 

the selection problem. Then, (4) has to be replaced by 



(11) ui(y) 



bOi(yc-y) + aOi for x < x c i 

bli(y-yc) + ali for x > x c i, bQi, bn > 0 



and the probability functions ki(x,y),, i = i , p. are now 

allowed to vary across populations. The Bayes utility for a 
random individual is defined analogous to (5) as Ri(x c ^). Let 
be the relative size of -mlation i. Then, when sampling 
randomly from the total population under consideration, the 
Bayes utility is equal to 



PS 
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P 



R(x cl 




= I siP^(xci). 



i=l 



But this is Just a weighted sum of the Bayes utilities of the 
separate populations. Hence, the decision procedure is 
optimal if. analogous to (5) to (8). for each population a 
separate Bayes rule is derived. 

An example of the model with real test data is given in 
Mellenbergh and van der Linden (1981). 



In the mastery decision problem, the unknown state of nature 
is the individual's true score on the criterion variable 
measured by the test. The data are the observed test scores. 
The true score is de'ined as the expected proportion of test 
items a given individual solves correctly. The true score of 
a random individual is denoted as T. with possible values 
t e [0.1]. A mastery level w ,c is assumed., and an individual 
is considered to master the criterion if t > t c . and not to 
master it otherwise. Unlike the selection problem, there i« 
no way whatsoever to measure the criterion Variable directly. 
Hence, a test model is needed to derive the statistical 
relation of X to the true score T. 

For the purpose of illustration, a threshold utility 
function is chosen. Although Figure 6 shows that this 
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function has a jump at t c that may be less realistic in some 



Insert Figure 6 about here 



applications, it has been studied extensively in the mastery 
testing literature (Hambleton & Novick. 1973; Huynh. 1976; 
Mellenbergh. Koppelaar, & van der Linden, 1977; van der 
Linden. 1980, 1982; Wilcox, 1977). The threshold utility 
function is defined by the following four constants 



(12) 



u(t) 



U00 
uiO 
uoi 
uil 



t < t C , X < x c 

t > tc. X < Xc 

t < tc. X > xc 

t > t C . X > x c 



However , since the derivation following below holds for any 
positive linear rescaling of (12), it will be assumed for 
convenience that uqq = u^ = 0. 

Let the joint distribut ion of X and T be given by the 
probability function k(x,t), and let g(t|x) s k(x.t)/h(x). 
For the sake of illustration, the optimal cut— off score x c 
will now be derived using a comparison of the posterior 
expected utilities for the mastery and nonmastery decision. 
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If the mastery decision is taken for a random individual 
with test score X = x. the posterior expected utility is 
equal to 

r 1 i^c 
(13) ExCuCtJlx) = u(t)g(t|x)dt = u 0 i g(t|x)dt 

J 0 J 0 
For the nonmastery decision 



(14) E 0 (U(T)|x) = \ u(t)g(t|x)dt = u 10 f 1 g(t|x)dt« 

J 0 J t c 



Suppose x were continuous. Then (13) and (14) would cross at 
the value of x for which 



ftc r 1 

u 01 J ST(t|x)dt = u 01 j g(t|x)dt 
0 t/» 



or 



ftc , J u 10 
(15) g(t|x)dt = 

J 0 



u 01 + u 10 



However. X is a discrete test score. Therefore, the Bayes 
utility is maximal if x c is chosen to be the smallest integer 
value larger than the solution to (15). 




A J0 
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It should be noted that this solution holds for any test 
model providing the probability function in the left-hand 
side of (15), and that it can be calculated only once such a 
model is specified. 

A usual choice in mastery testing is the beta— binomial 
model (Huynh, 1976; Mellenbergh, Koppelaar, & van der Linden, 
1977). In the model it is assumed that (1) the conditional 
distribution of X given T = t is the binomial, 

(16) f(x|t) = (£)t x (l-t) n - x , 

and (2) the marginal distribution of T is the beta 
distribution with probability density function 

(17) b(t) h B- 1 (v,w-n+l)t v - 1 (l-t) w - n . v > 0, w > n-1 , 

r l 

where B(v,w-n+l) 2 t v (l-t) w " n dt is the complete beta 

J 0 

function (e.g., Johnson & Kolz, 1970, chap. 24). The choice 
of (16) is motivated by the fact that the responses of a 
fixed individual to a series of test items can often be 
described as a sequence of Bernoulli trials, whereas (17) 
defines a flexible family of distributions on [0.1] that 
contains most true-score distributions occurring in practice. 
Keats and Lord (1962) have shown that moment estimators of v 
and w exist that are a simple function of Hx and the KR-21 
reliability coefficient. They also found a satisfactory fit 
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of the beta-binomial model to test score distributions 
ranging widely in form. 

For integer values of v and w, it holds that 



where f(-|t c ) is the binomial probability function with 
success parameter t c (Johnson & Kolz, 1970. sect. 24.6). 
Thus, a suitable estimate of the integral in (15) can be 
obtained via a table of the cumulative binomial. 

A Numerical Example 

Using the beta-binomial model, the result in (15) was used to 
calculate the optimal cut-off score x c for tests of length 
n = 20, mastery threshold t c = 14, and utility ratio 
u 10/ u 01 = * * data were simulated such that the average 

true sco-:e Hj and the KR-21 reliability coefficient varied 
systemati .lly (van der Linden. 1984). Table 1 shows how x c 
depends on these two parameters. For tests of high 



(18) 




1 f(Y|t c ) 



y=v+x 



Insert Table 1 about here 
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reliability x c varies hardly with jij, but this robustness is 
lost quickly for tests of lower reliability. For KR-21 = .05, 
[ly £ 13 yields xc £ 20, but the optimal cut-off score drops 
immediately to below zero for lij > 13. Only tests of high 
reliability yield stable cut-off scores. 

An unexpected phenomenon in Table 1 is the opposite 
direction in which x c varies with Hj. If the average true 
score ja*p goes down, xc goes up. Hence, for low performing 
populations the cut-off score should be set high, whereas it 
should be low for high performers. At first sight, this goes 
against our intuition. However, it is a logical consequence 
of our criterion of maximal expected utility. An analysis of 
(12) to (15) reveals the following: For high performing 
populations, almost all individuals are above the mastery 
threshold t c . Therefore, the Bayes utility of the decision 
procedure tends to consist only of contributions from true 
masters and false nonmasters. Since the proportion of true 
masters among these two categories depends on the cut-off 
score x c , and the utility of a true master typically is 
larger than the utility of a false nonmaster, x c will take a 
low value. This phenomenon was dubbed "the regress ion-from- 
the mean effect" in van der Linden (1980). It is a reversion 
of the well-known regress ion-to-the-mean effect due to the 
fact that inthe derivation of (15) the conclusion goes from 
the true score t c to x c instead of the other way around. 
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Placement Decisions with Normal-Ogive Utility 



The typical feature of the placement decision is the presence 
of more than one treatment and the fact that each individual 
is assigned to a treatment. As before, the unknown state cf 
nature is the individual's criterion score after a treatment. 
Hence, although there is one criterion common to all 
treatments, each individual has a different unknown state of 
nature for each criterion. 

Let Y be the criterion common to the treatments j = 1 , 
2, m, and let X denote the test score again. It is 

assumed that the treatments are ordered by the strict 
monotonicity conditions for the placement decision given in 
van der Linden (1981), so that an optimal cut— off score x-j 
exists for the decision between treatments j and j + 1 . For 
the sake of illustration, the normal— ogive utility function 
introduced by Novick and Lindl^y (1978; see also Berhold, 
1973) will be used (see also Figure 7): 



(19) uj(y) = 4> 



where Q> is the normal distribution function with parameters 
Hj and o j . 
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Insert Figure 7 about here 



The relevant quantity is the posterior expected utility 
for treatment j after test score X = x. This will be denoted 
as Ej(Uj(Y) |x) . It is assumed that the conditional 
distribution of Y given X = x is normal with conditional mean 
and variance E-j(Yjx) and Varj(Y|x), respectively. It follows 
that 



(20) Ej(uj(Y)|x) = <D 



Ej(Y|x) - Uj) 

[Var (Y|x) + o 2 ]K 
3 3 



The optimal cut-off score x-j is now the smallest value of x 
for which the posterior expected utility for treatment j + 1 
is larger than for treatment j, that is, the smallest value 
of x for which 



(21) <D 



Ej + i(Y|x) - n-j + 1 
[Var (Y|x) + a 2 



- CD \ 



Ej(Y|x) - ^ 

(Var (Y|x) + o 2 ]* 
3 3 



is positive. But this is also the smallest value of x for 
which 
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(22) 



E 3+1 (Yjx) - 

[Var (Y|x) + o 2 
3+1 3+1 



_ Ej(Y|x) - 

JX (Var (Y|x) + a 2 ] 1 ^ 
j 3 



is positive. If it is known how the posterior expectations 

and variances depend on x, the optimal cut— off score can be 

found graphically or numerically from (22). 

Suppose now that the posterior expectations are linear 

and that homoscedasticity may be assumed. That is, 

i 

(23) Ej(Y|x) = pjx + <Xj, j = 1 m 

and 

x = 0 n 

(24) Varj(Y|x) = Var-j(Y.X) 

j = 1 m 

where Var^Y.X) = { 1— [Cor j (Y, X) ] 2 }Varj ( Y) and Cor-j(Y.X) is 
the linear correction coefficient between Y and X for j. 
Substituting (23) and (24) into (22) yields as a result that 
Xj is the smallest value larger than 

(25) t £ j(Hj+r-<*j + i) - e j + l(Hj+a j )]/(ejpj +r -€ j + 1 pj) 



with 



6j = [Varj(Y.X) 
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More theory on placement decisions is given in van der 
Linden (1981). 

Quota-Restricted Placement 

In the above plcicement model, it was assumed that the number 
of vacant places in the treatments was unrestricted. In 
practice, however . quota restrictions regularly apply, and 
then a modification of the decision rule is necessary. 

Following Chuang. Chen and Novick (1981), three kinds of 
quota restrictions are distinguished: 

(1) Exactly N-; individuals should be assigned to treatment 
J n 

3 = 1 m. where £ Nj = N (number of examinees); 

3=1 

<2) At least Nj individuals should be assigned to treatment 

n 

j = 1 . . ... m. where Z Nj < N; 

0=1 

(3) At most Nj individuals should be assigned to treatment 

n 

j = 1 m. where £ Nj < N. 

3=1 

In a (strictly) monotone placement problem. the 
posterior expected utilites of adjacent pairs of treatments 
cross at most once in the range of test scores. Therefore , if 
the N examinees are arranged in decreasing order of test 
scores, the following placement rules maximize the Bayes 
utility in the above three cases: 

(1) Beginning with the highest scoring examinee. N m places 
in treatment m are filled, then N^^ places in treatment 
m-l . and so on . 



3C 
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( 2 ) Suppose the ( k-1 ) th examinee in the order o f exami nees 
has been assigned to treatment j. Then the following 
rule for k is optimal: 

(i) If Xfc > . assign k to treatment j. 

(ii) If xjj < x j~* anc * treatment j has not received Nj 
examinees, then choose from the following rules 
the one with the larger posterior expected 
utility: (a) assign k to j; (b) assign k to j-1 , 
reassigning the lowest scoring examinee in j+1 
to j ( and . if j + 1 then has fewer than N j + j 
examinees, the lowest scoring one in j+2 to j+1; 
etc. ) . 

(iii) If x^ < and j has received N-j examinees . 
then assign k to j-1 . 

(3) Again suppose the (k-l)th examinee has been assigned to 
treatment j . 

(i) If Xfc < then assign k to the treatment in 

{j-1. j-2 1) with the largest posterior 

expected utility. 

(ii) If x^ > and treatment j has received Nj 
examinees, then choose from the following rules 
the one with the larger posterior expected 
utility: (a) assign k to j-1; (b) assign k to j 
reassigning the highest scoring examinee in j to 
j+1 (and, if j+1 ha lready Nj + j examinees, the 
highest scoring one in j+1 to j+2; etc.) 
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(iii) If Xfc > Xj_i and j has not received Nj examinees, 
then assign k to j . 



Classification Decisions with Threshold Utility 

The same notation as for the problem of the placement 
decision will be used. However, because every treatment now 
has its own criterion, the criterion variable is treatment 
dependent and will be denoted as Yy 

Suppose that for each criterion a success threshold dj 
can be defined. Then the following threshold utility function 
may be a proper choice: 



(26) Uj<Yj> = 



wj for yi > dj 

vj for yi < d 3 



with 



w j > v j 3 = 1 m 



The last condition simply states that the utility of a 

b 1 



success on treatment g. is larger than the one of a failure. 



Let n*(dj|x) = 



gj(yj|x)dyj. where gj(Yj| x ) is the 
probability density function of Yj given X = x and treatment 
j. This quantity defines the probability of a failure on 
criterion Yj after treatment j. Van der Linden (1987) shows 
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that the following conditions are sufficient for a monotone 
Bayes rule in a classification problem with threshold 
utility: 

(27) w^j - Vj_! £ Wj - Vj j . 2 m 

(28) Qj(dj|x) is decreasing in x j = 1 m 

(29) Qj„ 1 (dj-l|x) -Qj(dj|x) is j = 2, m 

increasing in x 

The condition in (27) indicates that the relevarv order of 
the treatments is with respect to w j ~ v j ■ The other two 
conditions state that the probability of a failure should be 
decreasing in the test score, but that the difference between 
these probabilities for two adjacent treatments should 
increase . 

The optimal cut-off score for a decision between 
treatments j and j+1 is found by comparing posterior expected 
utilities. If an individual with test score X = x is assigned 
to treatment j, the posterior expected utility is equal to 

(30) Ej(u(Yj)|x) = Qj(dj|x)vj + (l-Oj(dj|x)]wj. 

The interest is in the value of x at which Ej(Uj(Yj)|x) and 
Ej+i(Uj + i ) |x) cross, that is. the value of x for which 
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Qj(dj|x)vj + [1-Qj(dj|x)]wj 



= Jlj + 1 (dj+l|x)vj+l + [l-^i+iCdj + l |x)]wj+l 



or 



(31) (Wj-vO^(dj|x) - (wj + l-vj + i)0; + 1 (dj + i|x) 



+ w 



>1 



= C 



However, since X is discrete, the optimal cut-off score is 
the smallest value of x for which the left-hand side of (31) 
is positive. The conditions in (27) to (29) guarantee that 
this expression is an increasing function of x. If it takes 
the value of zero outside the range of possible test scores, 
the cut-off score is set at its corresponding border. 

An interesting case arises if the utility parameters Wj 
and Vj do not vary across treatments. Then it follows that 
Wj - Vj = Wj +1 - v j + l and Wj + i - w j = 0. and the left-hand 
side of (31) reduces to 

(32) Qj(dj|x) - Q J+ i(dj+l|x) . 

This expression reminds us of (21). Analogous to the argument 
following (21),, it can be shown that for the choice of a 
nonnal distribution function for Qj(.|x) together with 
linear agression functions Ej(Y|x) = (JjX + ctj and homosce- 
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elasticity, an optimal value of Xj equal to the smallest value 
of x larger than 

< d i+l- a i+lHVari(Y.X)]X - (dj-cu) [Vari+1(Y.X)]H 

(33) — 

Pj+ltVarjCY.X)]* - [Var j+KY.X)]* 

is obtained (van der Linden, 1987). This quantity can easily 
be calculated if estimates of the regression parameters and 
the pooled variances of Y given X = x are available. 

Multivariate Test Scores 

The use of a test battery instead of a single test was 
mentioned earlier as a possible refinement of test-based 
decision making. How Bayes rules can be derived in such cases 
will now be shown for the problem of placement decisions. 

It is assumed that the preceding theory of placement 
decisions with threshold utility holds. How3ver, in addition 
to test score X, a second test score Z with information about 
the criterion variables Yj (j = l l( .... m) is given. Z is 
also assumed to be defined by number-right scoring and may 

take the values 0, 1 r. Let kj(x.yj.z) be the joint 

probability function of test scores X and Z and criterion 
score Yj. while p(z) is the probability function of Z. 
Further, let Sj denote the set of ordered pairs (x.z) for 
which treatment j is assigned . The Bayes utility when 
sampling from the population is defined as 
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m 

(34) R(S! S m ) = £ I 

3=1 S j 

It follows immediately that 

m 

(35) R(Sj S m ) =11 p(z) 

3=1 S j 

where q(x,yj|z) is the probability function of (X.Yj) given 
Z = z. Suppose now that (28) and (29) hold for each value of 
z. Then, for each possible z-coordinate . (35) can be 
maximized following (30) and (31) with respect to the x- 
coordinates in S j . Thus, the optimal sets Sj are defined by a 
series of (r+l)(m-l) cut-off scores namely m-1 scores for 
each of the r+1 possible values of z. 

By symmetry, it holds that the z-coordinates of the 
optimal sets Sj can also be defined by a series of cut— of f 

scores zj z m-l x * t * ie failure probabilities Qj(dj|z) 

have the properties in (28) and (29) for each value of x. In 
the testing literature, classification ruJes that are 
monotone in two distinct test scores are known as 
"conjunctive rules" . For classification into a given 
treatment such ruies require that the examinee passes a 
certain cut-off score on x as well as on y. In other words. 



Uj(yj)kj(x.yj.z)dyj 



Uj(yj)q(x,yj |z)dyj . 
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it is impossible to compensate a failure on one test by a 
high score on the other. 

Combination of Basic Decisions 

As noted before, a well-known example of combinations of 
basic decisions in education is an individualized instruction 
system. Figure 8 shows a simple system in which a selection 



Insert Figure 8 about here 



decision is followed by a module witl a mastery decision 
after which a placement decision guides the students through 
two possible sequences of mastery decisions. Real— life 
systems often have a more involved structure. 

In systems of more than one decision point, it is 
possible to optimize decision rules simultaneously. In doing 
su. L.ue efficient use of the data in the i5ystem can be made. 
Also, more realistic utility structures can be used. 

In order to illustrate the procedure,, the simple case of 
a system with one placement and one mastery decision (van der 
Linden & Vos. 1986, Vos & van der Linden. 1987) will be dealt 
with (see Figure 9). The placement test score is denoted as X 



Applications of Decision Theory 

42 



Insert Figure 9 about here 



(x = 0, 1, . . , m) the observed mastery test score as Y 

(y = 0, 1 n ), and the true mastery test score as T 

(t e [0,1]). Further, the following definitions are made: x c 
is a cut-off score on x, y c is a cut-off score on y, t c is 
the mastery threshold on t , 3 = 1, 2 are the two possible 
treatments and i = 0, 1 are the possible states of nonmastery 
(T < t c ) and mastery T > t c ), respectively. Without bothering 
about conditions for optimality, a moaotone rule with maximum 
utility is looked for; that is, from the pairs of cut— off 
scores (x c ,y c ), the one with the largest Bayes utility will 
be chosen , The following threshold utility function is 
adopted : 



(36) u i: (t) 



v00 + wj 
vlO + wj 
vOl + wj 
vi i + Wj 



for t < d, y < c 

for t > d, y < c 

for t < d, y > c 

for t > d (/ y > c, 



3 = 1, 2. 



The parameter v is dependent both on the mastery state and 
the mastery decision, but it is independent of the treatment. 
In addinion, the treatment-dependent parameter w can be used 
to allow for differences in , for instance , treatment costs . 
Although this function could be made more realistic by 
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replacing v by continuous functions of the true score t. it 
nicely demonstrates how in a simultaneous approach a utility 
function defined on the ultimate criterion of the system 
(mastery-nonmastery state) can be brought into a previous 
decision (placement decision). 

The Bayes utility for assignment to treatment 1 and a 
correct mastery decision is equal to 

Xc" 1 Yc- 1 *c 
(37) I I J (vOO+wl)(p(x.y.t)dt . 

x=0 y=0 0 

where (p(x.y.t) is the probability function of (X,Y,T). 
Combining the two possible treatments two possible mastery 
decisions and two possible mastery states, the total expected 
util ity when sampling trom the population of individuals is 
equal to the sum of eight expressions like (37). From van der 
Linden and Vos (1986) it can be verified that this sum can be 
reduced to the following posterior form: 



(38) R(x c .y c ) = constant + I [(-vQO-vil) (p 0 (t | y)+vii)K 0 (y) 



n 



Y=Yc 



0 



+ £ IvOOf [ {pi(t|x)-p 0 (t|x)|dt + wi - wQ)X(x) 
x=x c l J 0 J 
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m n [ f tc 
+ I X jtvoo+vil) rc 0 (t|x.y)dt " vii]rj 0 (x,y) 
x=x c y=y c l J 0 



- K v 00 + v ll> J^lCtjx.yJdt - v 11 ]Ti 1 (x,y)] } ,| 



with cpo^ly)- Pj(t|x), 7ij(t|x,y) being the posterior 
probability density functions of T given Y=y, X=x and 
{Y=y,X=x}. respectively, and where HjCx.y). Ji(y) and K-j(y) 
are the marginal probability functions. 

To obtain the optimal values of x c and y c , (38) has to 
be evaluated numerically for all possible pairs of values, 
whereupon the pair for which (38) is maximal can be 
determined. This can be done as soon as accurate estimates 
for the three different posterior probabilities of a failure 
in (38) are available. To obtain these estimates, a test 
theory model is needed. 



Conclusion 



A review of the applications of Bayesian decision theory to 
test-based decision making was given. Four basic types of 
decisions were distinguished, each of which may be subject to 
further restrictions or generalizations Decisions may also 
be made separately or in combination. The test theory 
literature contains many more results for selection and 
mastery decisions than was presented in this paper The 
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placement and classification decisions as well as some of the 
possible refinements and generalizations and the case of 
combinations of decisions have been largely unexplored. Of 
paramount importance, however, is research on realistic 
utility functions for test-based decisions . Such functions 
should not only yields robust decision rules (Vijn & 
Molenaar, 1981), but also be supported by empirical evidence 
of their appropriateness (Vrijhof, Mellenbergh & van den 
Brink. 1983). Developments in this field will be decisive for 
the applicability of Bayesian theory to test-based decision 
making. 
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Figure 2. Flowchart of a mastery decision 
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treatment 1 



treatment 2 




Figure 3 . Flowchart of a placement decision 
(case of two traatments) 
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Figure 4. Flowchart of a classification decision 
(case of two treatments) 
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Figure 6. Threshold utility function for a mastery decision 
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Figure 8. Example of an individualized study system 
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Figure 9, A system of one placement 
and one mastery decision 
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Table 1 \ 

Optimal Cut-off Scores for Populations Varying in Average 
True Score and Tests Varying in Reliability (n=20, t c =14, 

u io/ u oi =1 > 



KR-21 










Average 


Truo 


Score 










1 


3 


5 


7 


9 


11 


13 


15 


17 


19 


.95 


16 


15 


15 


15 


15 


15 


14 


14 


14 


14 


.80 


F 


19 


17 


16 


16 


15 


15 


14 


14 


14 


.65 


F 


F 


F 


F 


17 


16 


15 


14 


13 


12 


.50 


F 


F 


F 


19 


19 


17 


16 


13 


12 


P 


.35 


F 


F 


F 


F 


F 


20 


16 


13 


P 


P 


.20 


F 


F 


F 


F 


F 


F 


19 


11 


P 


P 


.05 


F 


F 


F 


F 


F 


F 


20 


P 


P 


P 


Note F 


indicates 


a cut 


-off 


score 


larger 


than 


20 







(all students fail); P indicates a cut-off score 
lower than 0 (all students pass). 
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